{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LAB 4b:  Create Keras DNN model.\n",
    "\n",
    "**Learning Objectives**\n",
    "\n",
    "1. Set CSV Columns, label column, and column defaults\n",
    "1. Make dataset of features and label from CSV files\n",
    "1. Create input layers for raw features\n",
    "1. Create feature columns for inputs\n",
    "1. Create DNN dense hidden layers and output layer\n",
    "1. Create custom evaluation metric\n",
    "1. Build DNN model tying all of the pieces together\n",
    "1. Train and evaluate\n",
    "\n",
    "\n",
    "## Introduction \n",
    "In this notebook, we'll be using Keras to create a DNN model to predict the weight of a baby before it is born.\n",
    "\n",
    "We'll start by defining the CSV column names, label column, and column defaults for our data inputs. Then, we'll construct a tf.data Dataset of features and the label from the CSV files and create inputs layers for the raw features. Next, we'll set up feature columns for the model inputs and build a deep neural network in Keras. We'll create a custom evaluation metric and build our DNN model. Finally, we'll train and evaluate our model.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in this student lab notebook -- try to complete this notebook first and then review the [solution notebook](../solutions/4b_keras_dnn_babyweight.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hJ7ByvoXzpVI"
   },
   "source": [
    "## Load necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import os\n",
    "import shutil\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set your bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = # REPLACE BY YOUR BUCKET\n",
    "\n",
    "os.environ['BUCKET'] = BUCKET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify CSV files exist\n",
    "\n",
    "In the seventh lab of this series [1b_prepare_data_babyweight](../solutions/1b_prepare_data_babyweight.ipynb), we sampled from BigQuery our train, eval, and test CSV files. Verify that they exist, otherwise go back to that lab and create them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DATA_PATH = \"gs://{bucket}/babyweight/data/train*.csv\".format(bucket=BUCKET)\n",
    "EVAL_DATA_PATH = \"gs://{bucket}/babyweight/data/eval*.csv\".format(bucket=BUCKET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage ls $TRAIN_DATA_PATH"   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage ls $EVAL_DATA_PATH"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set CSV Columns, label column, and column defaults.\n",
    "\n",
    "Now that we have verified that our CSV files exist, we need to set a few things that we will be using in our input function.\n",
    "* `CSV_COLUMNS` are going to be our header names of our columns. Make sure that they are in the same order as in the CSV files\n",
    "* `LABEL_COLUMN` is the header name of the column that is our label. We will need to know this to pop it from our features dictionary.\n",
    "* `DEFAULTS` is a list with the same length as `CSV_COLUMNS`, i.e. there is a default for each column in our CSVs. Each element is a list itself with the default value for that CSV column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine CSV, label, and key columns\n",
    "# Create list of string column headers, make sure order matches.\n",
    "CSV_COLUMNS = [\"weight_pounds\",\n",
    "               \"is_male\",\n",
    "               \"mother_age\",\n",
    "               \"plurality\",\n",
    "               \"gestation_weeks\"]\n",
    "\n",
    "# Add string name for label column\n",
    "LABEL_COLUMN = \"weight_pounds\"\n",
    "\n",
    "# Set default values for each CSV column as a list of lists.\n",
    "# Treat is_male and plurality as strings.\n",
    "DEFAULTS = [[0.0], [\"null\"], [0.0], [\"null\"], [0.0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make dataset of features and label from CSV files.\n",
    "\n",
    "Next, we will write an input_fn to read the data. Since we are reading from CSV files we can save ourself from trying to recreate the wheel and can use `tf.data.experimental.make_csv_dataset`. This will create a CSV dataset object. However we will need to divide the columns up into features and a label. We can do this by applying the map method to our dataset and popping our label column off of our dictionary of feature tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def features_and_labels(row_data):\n",
    "    \"\"\"Splits features and labels from feature dictionary.\n",
    "\n",
    "    Args:\n",
    "        row_data: Dictionary of CSV column names and tensor values.\n",
    "    Returns:\n",
    "        Dictionary of feature tensors and label tensor.\n",
    "    \"\"\"\n",
    "    label = row_data.pop(LABEL_COLUMN)\n",
    "\n",
    "    return row_data, label  # features, label\n",
    "\n",
    "\n",
    "def load_dataset(pattern, batch_size=1, mode='eval'):\n",
    "    \"\"\"Loads dataset using the tf.data API from CSV files.\n",
    "\n",
    "    Args:\n",
    "        pattern: str, file pattern to glob into list of files.\n",
    "        batch_size: int, the number of examples per batch.\n",
    "        mode: 'train' | 'eval' to determine if training or evaluating.\n",
    "    Returns:\n",
    "        `Dataset` object.\n",
    "    \"\"\"\n",
    "    # Make a CSV dataset\n",
    "    dataset = tf.data.experimental.make_csv_dataset(\n",
    "        file_pattern=pattern,\n",
    "        batch_size=batch_size,\n",
    "        column_names=CSV_COLUMNS,\n",
    "        column_defaults=DEFAULTS)\n",
    "\n",
    "    # Map dataset to features and label\n",
    "    dataset = dataset.map(map_func=features_and_labels)  # features, label\n",
    "\n",
    "    # Shuffle and repeat for training\n",
    "    if mode == 'train':\n",
    "        dataset = dataset.shuffle(buffer_size=1000).repeat()\n",
    "\n",
    "    # Take advantage of multi-threading; 1=AUTOTUNE\n",
    "    dataset = dataset.prefetch(buffer_size=1)\n",
    "\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create input layers for raw features.\n",
    "\n",
    "We'll need to get the data read in by our input function to our model function, but just how do we go about connecting the dots? We can use Keras input layers [(tf.Keras.layers.Input)](https://www.tensorflow.org/api_docs/python/tf/keras/Input) by defining:\n",
    "* shape: A shape tuple (integers), not including the batch size. For instance, shape=(32,) indicates that the expected input will be batches of 32-dimensional vectors. Elements of this tuple can be None; 'None' elements represent dimensions where the shape is not known.\n",
    "* name: An optional name string for the layer. Should be unique in a model (do not reuse the same name twice). It will be autogenerated if it isn't provided.\n",
    "* dtype: The data type expected by the input, as a string (float32, float64, int32...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_input_layers():\n",
    "    \"\"\"Creates dictionary of input layers for each feature.\n",
    "\n",
    "    Returns:\n",
    "        Dictionary of `tf.Keras.layers.Input` layers for each feature.\n",
    "    \"\"\"\n",
    "    inputs = {\n",
    "        colname: tf.keras.layers.Input(\n",
    "            name=colname, shape=(), dtype=\"float32\")\n",
    "        for colname in [\"mother_age\", \"gestation_weeks\"]}\n",
    "\n",
    "    inputs.update({\n",
    "        colname: tf.keras.layers.Input(\n",
    "            name=colname, shape=(), dtype=\"string\")\n",
    "        for colname in [\"is_male\", \"plurality\"]})\n",
    "\n",
    "    return inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create feature columns for inputs.\n",
    "\n",
    "Next, define the feature columns. `mother_age` and `gestation_weeks` should be numeric. The others, `is_male` and `plurality`, should be categorical. Remember, only dense feature columns can be inputs to a DNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def categorical_fc(name, values):\n",
    "    \"\"\"Helper function to wrap categorical feature by indicator column.\n",
    "\n",
    "    Args:\n",
    "        name: str, name of feature.\n",
    "        values: list, list of strings of categorical values.\n",
    "    Returns:\n",
    "        Indicator column of categorical feature.\n",
    "    \"\"\"\n",
    "    cat_column = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "            key=name, vocabulary_list=values)\n",
    "\n",
    "    return tf.feature_column.indicator_column(categorical_column=cat_column)\n",
    "\n",
    "\n",
    "def create_feature_columns():\n",
    "    \"\"\"Creates dictionary of feature columns from inputs.\n",
    "\n",
    "    Returns:\n",
    "        Dictionary of feature columns.\n",
    "    \"\"\"\n",
    "    feature_columns = {\n",
    "        colname : tf.feature_column.numeric_column(key=colname)\n",
    "           for colname in [\"mother_age\", \"gestation_weeks\"]\n",
    "    }\n",
    "\n",
    "    feature_columns[\"is_male\"] = categorical_fc(\n",
    "        \"is_male\", [\"True\", \"False\", \"Unknown\"])\n",
    "    feature_columns[\"plurality\"] = categorical_fc(\n",
    "        \"plurality\", [\"Single(1)\", \"Twins(2)\", \"Triplets(3)\",\n",
    "                      \"Quadruplets(4)\", \"Quintuplets(5)\", \"Multiple(2+)\"])\n",
    "\n",
    "    return feature_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create DNN dense hidden layers and output layer.\n",
    "\n",
    "So we've figured out how to get our inputs ready for machine learning but now we need to connect them to our desired output. Our model architecture is what links the two together. Let's create some hidden dense layers beginning with our inputs and end with a dense output layer. This is regression so make sure the output layer activation is correct and that the shape is right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_outputs(inputs):\n",
    "    \"\"\"Creates model architecture and returns outputs.\n",
    "\n",
    "    Args:\n",
    "        inputs: Dense tensor used as inputs to model.\n",
    "    Returns:\n",
    "        Dense tensor output from the model.\n",
    "    \"\"\"\n",
    "    # Create two hidden layers of [64, 32] just in like the BQML DNN\n",
    "    h1 = tf.keras.layers.Dense(64, activation=\"relu\", name=\"h1\")(inputs)\n",
    "    h2 = tf.keras.layers.Dense(32, activation=\"relu\", name=\"h2\")(h1)\n",
    "\n",
    "    # Final output is a linear activation because this is regression\n",
    "    output = tf.keras.layers.Dense(\n",
    "        units=1, activation=\"linear\", name=\"weight\")(h2)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create custom evaluation metric.\n",
    "\n",
    "We want to make sure that we have some useful way to measure model performance for us. Since this is regression, we would like to know the RMSE of the model on our evaluation dataset, however, this does not exist as a standard evaluation metric, so we'll have to create our own by using the true and predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rmse(y_true, y_pred):\n",
    "    \"\"\"Calculates RMSE evaluation metric.\n",
    "\n",
    "    Args:\n",
    "        y_true: tensor, true labels.\n",
    "        y_pred: tensor, predicted labels.\n",
    "    Returns:\n",
    "        Tensor with value of RMSE between true and predicted labels.\n",
    "    \"\"\"\n",
    "    return tf.sqrt(tf.reduce_mean((y_pred - y_true) ** 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build DNN model tying all of the pieces together.\n",
    "\n",
    "Excellent! We've assembled all of the pieces, now we just need to tie them all together into a Keras Model. This is a simple feedforward model with no branching, side inputs, etc. so we could have used Keras' Sequential Model API but just for fun we're going to use Keras' Functional Model API. Here we will build the model using [tf.keras.models.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) giving our inputs and outputs and then compile our model with an optimizer, a loss function, and evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_dnn_model():\n",
    "    \"\"\"Builds simple DNN using Keras Functional API.\n",
    "\n",
    "    Returns:\n",
    "        `tf.keras.models.Model` object.\n",
    "    \"\"\"\n",
    "    # Create input layer\n",
    "    inputs = create_input_layers()\n",
    "\n",
    "    # Create feature columns\n",
    "    feature_columns = create_feature_columns()\n",
    "\n",
    "    # The constructor for DenseFeatures takes a list of numeric columns\n",
    "    # The Functional API in Keras requires: LayerConstructor()(inputs)\n",
    "    dnn_inputs = tf.keras.layers.DenseFeatures(\n",
    "        feature_columns=feature_columns.values())(inputs)\n",
    "\n",
    "    # Get output of model given inputs\n",
    "    output = get_model_outputs(dnn_inputs)\n",
    "\n",
    "    # Build model and compile it all together\n",
    "    model = tf.keras.models.Model(inputs=inputs, outputs=output)\n",
    "    model.compile(optimizer=\"adam\", loss=\"mse\", metrics=[rmse, \"mse\"])\n",
    "\n",
    "    return model\n",
    "\n",
    "print(\"Here is our DNN architecture so far:\\n\")\n",
    "model = build_dnn_model()\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the DNN using the Keras plot_model utility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.keras.utils.plot_model(\n",
    "    model=model, to_file=\"dnn_model.png\", show_shapes=False, rankdir=\"LR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run and evaluate model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and evaluate.\n",
    "\n",
    "We've built our Keras model using our inputs from our CSV files and the architecture we designed. Let's now run our model by training our model parameters and periodically running an evaluation to track how well we are doing on outside data as training goes on. We'll need to load both our train and eval datasets and send those to our model through the fit method. Make sure you have the right pattern, batch size, and mode when loading the data. Also, don't forget to add the callback to TensorBoard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_BATCH_SIZE = 32\n",
    "NUM_TRAIN_EXAMPLES = 10000 * 5  # training dataset repeats, it'll wrap around\n",
    "NUM_EVALS = 5  # how many times to evaluate\n",
    "# Enough to get a reasonable sample, but not so much that it slows down\n",
    "NUM_EVAL_EXAMPLES = 10000\n",
    "\n",
    "trainds = load_dataset(\n",
    "    pattern=TRAIN_DATA_PATH,\n",
    "    batch_size=TRAIN_BATCH_SIZE,\n",
    "    mode='train')\n",
    "\n",
    "evalds = load_dataset(\n",
    "    pattern=EVAL_DATA_PATH,\n",
    "    batch_size=1000,\n",
    "    mode='eval').take(count=NUM_EVAL_EXAMPLES // 1000)\n",
    "\n",
    "steps_per_epoch = NUM_TRAIN_EXAMPLES // (TRAIN_BATCH_SIZE * NUM_EVALS)\n",
    "\n",
    "logdir = os.path.join(\n",
    "    \"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n",
    "tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
    "    log_dir=logdir, histogram_freq=1)\n",
    "\n",
    "history = model.fit(\n",
    "    trainds,\n",
    "    validation_data=evalds,\n",
    "    epochs=NUM_EVALS,\n",
    "    steps_per_epoch=steps_per_epoch,\n",
    "    callbacks=[tensorboard_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize loss curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot\n",
    "import matplotlib.pyplot as plt\n",
    "nrows = 1\n",
    "ncols = 2\n",
    "fig = plt.figure(figsize=(10, 5))\n",
    "\n",
    "for idx, key in enumerate([\"loss\", \"rmse\"]):\n",
    "    ax = fig.add_subplot(nrows, ncols, idx+1)\n",
    "    plt.plot(history.history[key])\n",
    "    plt.plot(history.history[\"val_{}\".format(key)])\n",
    "    plt.title(\"model {}\".format(key))\n",
    "    plt.ylabel(key)\n",
    "    plt.xlabel(\"epoch\")\n",
    "    plt.legend([\"train\", \"validation\"], loc=\"upper left\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "OUTPUT_DIR = \"babyweight_trained\"\n",
    "shutil.rmtree(OUTPUT_DIR, ignore_errors=True)\n",
    "EXPORT_PATH = os.path.join(\n",
    "    OUTPUT_DIR, datetime.datetime.now().strftime(\"%Y%m%d%H%M%S\"))\n",
    "tf.saved_model.save(\n",
    "    obj=model, export_dir=EXPORT_PATH)  # with default serving function\n",
    "print(\"Exported trained model to {}\".format(EXPORT_PATH))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls $EXPORT_PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab Summary: \n",
    "In this lab, we started by defining the CSV column names, label column, and column defaults for our data inputs. Then, we constructed a tf.data Dataset of features and the label from the CSV files and created inputs layers for the raw features. Next, we set up feature columns for the model inputs and built a deep neural network in Keras. We created a custom evaluation metric and built our DNN model. Finally, we trained and evaluated our model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-1.m68",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-1:m68"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
